The present invention relates to an image sensing apparatus, having an image stabilization function, capable of obtaining a high-resolution image by slightly shifting an image formation position by an optical system, on an image sensing device for performing photo-electric conversion on the image, and an image synthesis apparatus for synthesizing images obtained by the image sensing apparatuses.
An electronic still camera which uses a solid-state image sensing device, such as a charge coupled device (CCD), instead of a silver-halide film has been commercialized. An electronic still camera is superior to a camera using the silver-halide film in instantaneity, however, inferior in resolution and dynamic range.
In order to improve the resolution, which is one of the above defects of an electronic still camera, an image sensing apparatus adopting pixel shifting method has been proposed. In the pixel shifting method, a plurality of images are sensed while slightly shifting the image formation position of an image, incoming through an optical system, on an image sensing device for performing photo-electric conversion on the image, and the plurality of sensed images are synthesized using a predetermined method to obtain a single high-resolution image. As the prior arts of the pixel shifting method, there are following Japanese Patent Application Laid-Opens, for example.
In the Japanese Patent Application Laid-Open No. 60-27278, a wedge-shaped prism provided in front of the lens systems is rotated about the optical axis, thereby the light path of an optical image formed on an image sensing device is shifted in parallel to the optical axis of the lens system. Then, output images are synthesized to obtain a single high-resolution image.
In the Japanese Patent Application Laid-Open No. 60-91774, in an optical system configured with a magnification optical sub-system and a master optical system, a part of the lens of the master optical system is shifted in the vertical direction with respect to the optical axis, thereby the light path of the optical image formed on an image sensing device is shifted in parallel to an optical axis of the optical system. Then the obtained output images are synthesized to obtain a single high-resolution image.
Further, in the Japanese Patent Application Laid-Open No. 61-236282, a transparent plane parallel plate provided in front of an image sensing device is rotated about an axis which is perpendicular to the optical axis of the image sensing device, thereby the light path of the optical image formed on the image sensing device is shifted in parallel to the optical axis. Then the output images are synthesized to obtain a single high-resolution image.
In the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), a variable apical angle prism provided in front of an optical system is operated on the basis of a vibration signal and a pixel shifting signal, thereby shifting the light path of the optical image formed on an image sensing device in parallel to the optical axis. Accordingly, compensation of vibration caused by a user as well as improvement of resolution of an image are achieved simultaneously.
Note, in this specification, any unintentional undesired movement of camera is expressed as xe2x80x9cvibrationxe2x80x9d, and the movement is not limited to periodic motion.
However, in the aforesaid pixel shifting methods, it takes a long time from the first image signal until the last image signal are obtained, similarly to the case of performing multiple exposure in a still camera. Thus, when the electric still camera is vibrated, the quality of an image decreases, namely, a high-resolution image may not be obtained by performing pixel shifting. Thus, in order to overcome this problem, there are following Japanese Patent Application Laid-Opens, for example.
In the Japanese Patent Application Laid-Open No. 7-240932 (U.S. patent application Ser. No. 08/391,388), using a variable apical angle prism provided in front of the optical system or a moving lens system provided behind the optical system, both compensation of vibration and improvement of resolution of the image are achieved at the same time.
Further, according to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), because resolution in pixel shifting operation decreases when the focal length of the optical system is larger than a predetermined value, pixel shifting is disabled in such a case.
Further, in order to widens dynamic range, which is the other defect of the electronic still camera, there are the following Japanese Patent Application Laid-Opens.
In the Japanese Patent Application Laid-Open No. 1-319370, an image sensing device is exposed a plurality of times with different luminous exposures, and a plurality of images obtained under this operation are synthesized to form a signal image of wide dynamic range.
In the Japanese Patent Application Laid-Open No. 7-264488, a plurality of image sensing devices having different sensibilities are used, and a plurality of images obtained by these image sensing devices are synthesized to form a single image of wide dynamic range.
Furthermore, as a technique for overcoming the aforesaid two problems at the same time, the Japanese Patent Application Laid-Open No. 8-37628 (U.S. patent application Ser. No. 08/505,608) discloses that at least one of a plurality of images obtained while performing pixel shifting is sensed in different luminous exposure from luminous exposure used for sensing other images, thereby obtaining an image of high resolution and wide dynamic range.
Further, in an image sensing apparatus having an image sensing device, it is possible to determine vibration of an image in advance to actually sensing the image by obtaining a movement vector of the image from time-sequential outputs from the image sensing device. Accordingly, in the Japanese Patent Application Laid-Open No. 2-57078 as a prior art in this field, a movement vector of an image is detected continuously, and, when the movement vector becomes the smallest, the image sensed at that time is decided as an image to be recorded, thereby reducing the effect of the vibration of an image sensing apparatus during exposure.
Furthermore, in the Japanese Patent Application Laid-Open No. 8-172568, movement vectors between a plurality of images sensed while performing pixel shifting are obtained, and component of vibration due to vibration of an image sensing apparatus or of an object are removed by performing interpolation, thereafter, the images are synthesized to form a single image of high resolution.
However, in the aforesaid conventional examples disclosed in the Japanese Patent Application Laid-Opens, there are following defeats.
In the methods disclosed in the Japanese Patent Application Laid-Open Nos. 60-27278, 60-91774, and 61-236282, no vibration correction mechanism is provided. Therefore, it is not possible to obtain a high-resolution image when blurring of an image caused by vibration of an image sensing apparatus is large. This is because the operation for obtaining a plurality of images in the pixel shifting method is the same as that of multiple exposure, as described above, and the time required in these operations, namely the time when the first image is sensed until the last image is sensed, is longer than the time required for performing a normal image sensing operation. As a result, effects of vibration on an image is greater when performing pixel shifting operation than when performing the normal image sensing operation vibration.
Further, according to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), the variable apical angle prism, which is the light path shifting means used for vibration compensation and pixel shifting operation, is provided in front of the optical system, therefore, the coefficient (vibration compensation coefficient) for converting a vibration signal into a value for operating the variable apical angle prism does not change in response to zooming operation. Accordingly, when performing zooming operation, only the coefficient (pixel shifting coefficient) for converting a pixel shifting signal into a value for operating the variable apical angle prism needs to be changed. However, when the light path shifting means is provided in the middle of the optical system in order to down-sizing the optical system, it is necessary to convert both the vibration signal and the pixel shifting signal into values using specific coefficients when performing zooming operation, and operate the light path shifting means in accordance with the values, but there is no disclosure on such the conversion and operation in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407).
Furthermore, in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), a pixel shifting mechanism is applied to a video camera (camcorder) for recording a moving image, therefore, the interval for taking images is fixed to the field frequency of a moving image. However, when the pixel shifting mechanism is applied to a so-called electronic still camera for recording a still image, it is advantageous to use an image sensing device whose image-taking interval can be changed on the basis of the charging period of the image sensing device (i.e., luminance of an object), since the camera can sense an object in a wide luminance range. In this case, as the image-taking interval becomes longer, effects of vibration on the image becomes stronger, therefore, delicate control of the pixel shifting operation in accordance with image sensing conditions is required. However, in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), only change in pixel shifting operation in accordance with a focal length of the optical system is disclosed. In addition, there is no teaching on dynamic range expansion.
In the Japanese Patent Application Laid-Open No. 7-240932 (U.S. patent application Ser. No. 08/391,388), pixel shifting is performed even when the resolution of vibration compensation is not good or vibration compensation has failed. As a result, a high-resolution image is not obtained; on the contrary, the quality of the image obtained by performing pixel shifting operation would be lower than an image obtained without performing pixel shifting operation.
According to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), under conditions in which it is predicted that resolution higher than a predetermined level can not be obtained by performing pixel shifting operation, the pixel shifting operation is disabled. However, the prediction is not performed on the basis of an actual effect of vibration on an image. Therefore, similarly to the cases of other references as explained above, when an image sensing apparatus vibrates by a large displacement amount, the obtained image would have a lower quality than an image obtained without performing pixel shifting operation.
Further, according to the Japanese Patent Application Laid-Open No. 60-91774, resolution of an image is increased, however, there is no teaching on dynamic range expansion.
On the contrary, in the Japanese Patent Application Laid-Open Nos. 1-319370 and 7-264488, dynamic range expansion is explained, however, how to increase resolution of an image is not discussed.
Whereas, in the Japanese Patent Application Laid-Open No. 8-37628 (U.S. patent application Ser. No. 08/505,608), methods for increasing resolution of an image and widening dynamic range are disclosed, however, there is no detailed description on method for determining luminous exposures to be used for sensing a plurality of images for dynamic range expansion. Therefore, the disclosed method is not possible to delicately cope with various scenes which have various luminous distributions. Further, pixel shifting operation requires longer time for performing exposing operation, similarly to a case of performing multiple exposure operation, as described above, therefore, it is necessary to cope with vibration problem. However, there is no teaching on any technique for overcoming the vibration problem.
Further, in the Japanese Patent Application Laid-Open No. 2-57078, there is no teaching on pixel shifting operation, therefore, only reduction of effect of vibration on an image sensed in a normal image sensing operation is achieved. Thus, improvement in resolution of an image is not expected. Furthermore, blurring due to a movement of an object can not be reduced.
Further, in a case of performing pixel shifting operation to increase resolution of an image, it is necessary to control the shift amount to be a predetermined amount based on the interval between pixels. However, there is no teaching on optical vibration compensation means using, e.g., a variable apical angle prism in the Japanese Patent Application Laid-Open No. 8-172568, therefore, blurring of an image while performing pixel shifting operation is large and occurs at random. Therefore, there is no guarantee that pixel shifting by the predetermined shift amount is always performed. In addition, even though blurring of an object is corrected by performing interpolation, possibility of obtaining a high resolution image is low. Furthermore, there is no teaching on warning a user of an image not being obtained in desired resolution, nor about an alternative suggestion to be followed for improving resolution of the image.
The present invention has been made in consideration of the above situation, and has as its object to provide an image sensing apparatus and method capable of performing image stabilization and pixel shifting simultaneously and at high resolution regardless of the configuration of the optical system and the configuration of image stabilization function.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals; shifting means for shifting an image formation position of the optical image formed on the image sensing means to a plurality of different positions; first vibration detection means for detecting vibration of the image sensing apparatus and outputting vibration information; control means for controlling the shifting means on the basis of the vibration information outputted by the first vibration detection means; and image signal synthesis means for synthesizing image signals of a plurality of images outputted by the image sensing means to generate a single image, wherein the image sensing means converts the optical image into electric signals at each of the plurality of different image formation positions shifted by the shifting means.
It is another object of the present invention to provide an image sensing apparatus capable of reducing the effect of movement of an object while performing pixel shifting so as to obtain an image of high resolution.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals; image sensing control means for controlling the image sensing means to sense a plurality of images within a predetermined period of time; image signal synthesis means for synthesizing image signals of the plurality of images outputted by the image sensing means to generate a single image; division means for dividing an image into a plurality of small areas; relationship determination means for determining relationship between the plurality of images by each of the plurality of small areas; and image synthesis control means for controlling image synthesis operation by the image signal synthesis means on the basis of the relationship between the plurality of images determined by the relationship determination means.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.